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Abstract: The objectives of this study were to evaluate the perceived student value 
of topics taught in Butler University’s Introduction to Experiential Rotations 
(RX500) course, implement course revisions to address any perceived 
weaknesses, and to reassess the course following implementation of those course 
revisions. Advanced Pharmacy Practice Experience (APPE) students from Butler 
University’s 2012 doctorate of pharmacy class were initially surveyed to assess 
the perceived usefulness and design of RX500. Based on the findings of the initial 
data, course revisions were developed and implemented for the following 
semester’s RX500 course. In order to assess for potential changes in the 
perceived value of the course following implementation, a follow-up survey was 
sent to students completing the revised course in the fall of 2012. Fifty-eight 
students completed the initial survey in the fall of 2011, and 34 completed the 
follow-up survey in the fall of 2012. The majority of students in both groups had 
completed at least four APPE experiences at the time of survey completion. Both 
of the study groups rated hands-on activities and visual presentations as their 
most beneficial learning styles within the classroom setting. Based on the initial 
survey findings, two topics were removed from the course, two topics were revised 
within the course, and three topics were added to the course. The follow-up 
survey results indicated an increased perceived usefulness of the two revised 
topics; however, the three new topics added to the course had mixed results. 
Faculty can continue to work on delivery of newer topics being added to the 
APPE preparation course. 

Keywords: course revision, health profession curriculum, clinical rotation 
preparation, student perceptions 


Background 

Student, faculty, and preceptor perceptions have been used to shape and refine curricular 
structure within colleges of pharmacy nationwide (Ho, 2009; Reid, 2002; Scott, 2010; Stevenson, 
2011). Additionally, the Accreditation Council for Pharmacy Education (ACPE) 2011 guideline 
22.2 recommends utilizing student perspectives in evaluation of curriculum, achievement of 
competencies, and other aspects of the professional program in the form of course evaluations, 
focus groups, or surveys (ACPE 2011). Traditional coursework within the doctor of phannacy 
curriculum includes didactic learning in the classroom setting during the first three years of the 
professional school, followed by experiential learning throughout the final year of the program. 
During that experiential learning year, all pharmacy students are required to complete at least 
1440 hours of Advanced Pharmacy Practice Experience (APPE). These experiences are designed 
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to provide a wide exposure to various patients and disease states, highlighting the impact of 
phannacists in many different settings. This final year is often considered a capstone year and an 
opportunity to put to practice knowledge gained during the previous curricular years. Two 
noteworthy publications within pharmacy education literature illustrate use of a survey tool 
called the Perceptions of Preparedness (PREP) survey (Reid, 2002 & Scott, 2010) to assess 
student perceptions of preparedness prior to these clinical APPE. The overarching goal of the 
PREP survey was to utilize information gained to improve the pharmacy program curriculum 
prior to APPE. 

The first instance of the PREP survey was published by Reid, et al. (2002) at the 
University of Florida. Pharmacy faculty created 41 competencies essential for completion prior 
to the APPE year. These competencies were used to develop the PREP survey. Student 
perceptions of confidence in each of these competencies were evaluated using a seven-point 
Likert-scale. To evaluate the results, investigators divided the competencies into five factors: 
administrative, communication, psychosocial, research, and technical. The study was conducted 
for four years from 1997 to 2001. Student responses revealed that students were most confident 
with the communication, psychosocial, and technical competencies, while students reported 
being least confident with the administrative and research competencies. 

Similarly, Scott (2010) and colleagues at North Dakota State University published results 
of a modified PREP survey (some competencies that were not relevant to the pharmacy program 
were removed). Students were asked to rate confidence using a five-point Likert-scale, and 
responses were assessed using the same five factors. The surveys were administered for four 
years from 2004 to 2008. Students at North Dakota State University were most confident with 
the communication and psychosocial competencies, while they expressed lower confidence in 
the administrative, research, and technical competencies. 

Not surprisingly, both studies found that graduating students become more confident as 
they progressed through the phannacy program and that student confidence in a given area 
increased following completion of a course that addressed that specific competency (Reid, 2002; 
Scott, 2010). Additionally, assessments of students from different graduating classes at the same 
point in the curriculum (e.g. comparing first year pharmacy students from each year of the study) 
revealed increasing confidence year by year. According to the authors from both studies, this 
demonstrated the positive impact of curriculum changes made during the four years of the study. 

Outside of the PREP surveys, Ho et al. (2009) published positive student perceptions 
following a curricular overhaul. The Chinese University of Hong Kong School of Phannacy 
restructured and revised their curriculum by planning specific learning outcomes for their 
graduates. The initial curriculum restructure was developed by the curriculum committee with 
revisions and implementation done by all faculty members. In addition to course specific student 
focus group feedback, perceptions were gathered via the graduate student survey. Students were 
asked to evaluate 13 curricular competencies, using a five-point Likert scale. This survey was 
administered annually within 6 months after graduation. Data was collected over a four-year 
timespan, starting with the old curriculum and finishing with two years of new curriculum 
graduates. One hundred eleven graduates (95%) completed the survey and significant 
improvement was noted in 10 of 13 professional and generic competencies, including critical 
thinking and interpersonal skills. No statistically significant improvement was noted in computer 
literacy, problem solving, or self-managed learning. The authors noted overall success in the 
implementation of their outcomes based curriculum, utilizing student perceptions as their 
primary method of assessment. 
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These studies all demonstrate the use of survey-based studies of students’ perceived value 
as an assessment tool when restructuring a pharmacy curriculum. However, the aforementioned 
studies outlined changes to an entire curriculum and tracked students throughout four years; the 
goal of this current study was to capture a snapshot of a single course. 

The nearest course specific revision comes from Auburn University Harrison School of 
Phannacy (AUHSP) (Stevenson, 2011). As a subset of continuous quality improvement within 
the curriculum, the APPE were reviewed by an experiential subcommittee, built within the larger 
curricular committee. Prior to assessment of the APPE, materials were collected, including 
syllabi, orientation materials, and course evaluations. Additionally, a 22 item survey was sent to 
fourth year pharmacy students completing rotations during the last academic month of their 
professional program. The survey questions were derived from concepts outlined in the 
American College of Clinical Pharmacy experiential education White Paper and Position 
Statement (Haase, 2008a; Haase, 2008b). While the number of students completing the survey is 
not reported, the authors report that student responses were positive when asked about exposure 
to multiple patient populations and disease states. Students also positively reported opportunities 
for collaboration, self-directed learning, and access to drug information resources. Students did 
identify areas for improvement, including additional electives, clearly defined objectives, and 
provision of constructive feedback. In conjunction with other materials gathered and faculty 
focus sessions, specific areas for improvement in AUHSP’s APPE were reported. These areas 
included: further identification of elective opportunities, increased use of the standardized 
evaluation form, and consistent use of syllabi templates with incorporation of clear objectives. 
The authors demonstrate effective use of student feedback in restructuring APPE at AUHSP. 

Based on a review of current literature, this study appears to be the first research 
evaluating the impact of student perceptions on a single didactic course within a college of 
phannacy. This study was conducted at Butler University, a private university located on the 
north side of Indianapolis, Indiana (Butler University, 2012). Housed within Butler University’s 
College of Pharmacy and Health Sciences (COPHS), the Doctor of Pharmacy program is 
comprised of two, liberal-arts based pre-professional phase years and four professional phase 
years. COPHS students take part in 10, 4-week APPE during the fourth professional year. 
Clinical sites for APPE include statewide health care facilities, as well as opportunities for travel 
to national and international sites. 

The Introduction to Experiential Rotations (RX500) is a required course within the 
COPHS pharmacy curriculum (Butler University, 2012). This one-credit hour course is taken 
during the spring semester of the third professional year, prior to beginning APPE. RX500 is 
primarily lecture-based and is taught by various university faculty and guest lecturers. Student 
assessments and grades are determined by attendance, assignments, and two examinations 
focused on 200 important medications. 

The goals of the RX500 course are as follows (RX500, 2011): 

1. To introduce students to, and prepare them for, the concept and practice requirements of 
the Advanced Pharmacy Practice Experiences (APPE). 

2. To facilitate scheduling of rotations and ensure completion of necessary forms, 
certifications and tests that need to be completed prior to starting rotations. 

3. To make students aware of the guidelines and expectations for [required seminars during 
the APPE year]. 

Traditional topics discussed throughout the semester range from practical skills (e.g. 
PowerPoint presentation tips) to reviews of clinical information (e.g. geriatrics overview) (see 
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Appendix 1 for full topic list). Topics covered in this course are those not covered in other areas 
of the PharmD curriculum. The majority of topics are covered within a one-hour course meeting; 
however, some topics are covered in only a small portion of the 50-minute class meeting, while 
others span over the course of two 50-minute class sessions. 

At the time of project conception, COPHS faculty and administration were re-assessing 
the Doctor of Pharmacy curriculum. Although RX500 had been in existence for over ten years, 
its content and structure had been largely unrevised since its inception. As a result, this course 
was identified as one that might potentially benefit from revisions. Because the course is largely 
focused on student preparation for the APPE year, the goal was to understand student needs in 
terms of APPE preparation and to revise the course accordingly. This study utilized APPE 
student perceptions of RX500 to change the content and delivery of information within the 
course. 


Objectives 


The objectives of this study were: 

• To assess the perceived student value of the topics within the RX500 course 

• To implement course revisions to address any perceived course weaknesses 

• To reassess the course following completion of course revision implementation 

Methods 

The study was designed to be completed in three phases: an initial survey phase, an 
implementation phase, and a follow-up survey phase (see Figure 1). These three phases are 
outlined in detail below. All research and study design was approved by the Butler University 
Institutional Review Board. 


Fall 2011 Fall 2012 



Implementation of 
course revisions 

(Class 2013) 


Figure 1. Study timeline. 

Initial Survey Methods 

All COPHS Doctor of Pharmacy students, most recently completing the RX500 course in 
the spring of 2011 (the Pharmacy Class of 2012), were invited via email to participate in the 
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study. Email invitations were sent in September of 2011 and included a link for survey 
completion via SurveyMonkey™(^/y?e«t/z'x /). Students were not given an incentive for 
participating in the study but were asked to participate in order to assist in future course revision. 

The survey first asked students to rate the spring 2011 RX500 topics as “Not Useful,” 
“Useful, Needs Improvement,” or “Useful As Is.” The second question assessed student opinions 
on the need for additional topics. Students were allowed to select from a list of potential topics 
generated from ACPE Pre-APPE Perfonnance Domains and Abilities and from other phannacy 
education literature, or they were allowed to provide open-ended responses in a blank space 
provided (American Council for Pharmacy Education, 2011). Students were then asked to 
provide infonnation on their learning style, number and type of rotations completed to date, and 
previous phannacy work experience. This infonnation was gathered in order to assess any 
potential differences between the pre-intervention and post-intervention groups. 

Implementation Methods 

Initial survey results were compiled and analyzed prior to the spring 2012 course. Topics 
were identified to remove, revise, or add to the course. All decisions were made in conjunction 
with RX500 faculty course coordinators. 

Follow-Up Survey Methods 

Students in the Class of 2013 were given a similar SurveyMonkey™ survey in September 
2012 to assess potential changes in perceived value of the course. The survey was modified to 
account for changes to the spring 2012 RX 500 topic list. 

Statistical Analysis 

Data were initially summarized using SPSS software, version 19.0 (IBM Corp., Somers, 
New York). Descriptive statistics were utilized to evaluate the results from the 2011 and 2012 
surveys individually. For topics revised in 2012, survey results from initial and follow-up 
surveys were analyzed using Mann-Whitney statistical analysis. A positive change in perception 
was viewed as a majority of responses indicating progression through the three categories (from 
Not Useful to Useful, Needs Improvement, from Useful, Needs Improvement to Useful As Is, 
etc.), and vice versa. 


Results 


Study Population 

A total of 58 out of 116 (50%) students from the class of 2012 completed the initial 
survey, compared to 34 out of 118 (28.8%) students from the class of 2013 in the follow-up 
survey period. The majority of students in the initial and follow-up surveys had completed at 
least 4 APPE experiences at the time of survey completion (100% and 91%, respectively). 
Additionally, the majority of students (>60%) in both groups had completed the required 
community practice and general medicine experiences at the time of survey completion. More 
students from the initial survey had completed an institutional practice rotation (72.4%) as 
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compared to students from the follow-up survey (26.5%). The initial survey group and follow-up 
survey group had similar rates of prior pharmacy work experience (98.3% and 94.1%, 
respectively) with the majority of students (82.3% in initial group, 84.4% in follow-up group) 
working in a retail setting. In both of the study groups, hands-on activities (53.4% in initial, 
55.9% in follow-up) and visual presentation (37.9% in initial, 38.2% in follow-up) were the 
learning styles most frequently indicated as most beneficial. 

Initial Survey 

In order to identify topics to remove for the spring 2012 course, student ratings from 
survey question number one were assessed (Table 1). The top three topics receiving a rating of 
Not Useful were: Code Response (23.3%), Professionalism (21.7%), and Practice Scenarios 1 & 
2 (20.0%). As a result, the Code Response and Practice Scenarios lectures were removed from 
the 2012 course. Although some students did not see the value in the Professionalism topic, 
course coordinators felt that it was a subject necessary for APPE preparation and helped to fulfill 
the course objective of setting “guidelines and expectations.” 

In terms of lectures needing revision, student responses of Useful, Needs Improvement 
from question number one were identified. The top three topics receiving this rating were: 
PhannD Project Proposal/Abstract/Manuscript (45.0%), Medication Therapy Management 
(MTM) (43.3%), and Curriculum Vitae (CV) Tune-Up (38.3%). In response to the survey 
results, all three of these topics were modified; the PhannD Project and Curriculum Vitae topics 
were revised within the course, while the MTM lecture was moved to a community phannacy 
seminar series taken during the students’ APPE year. In delivering these revised topics, faculty 
were encouraged to keep in mind that the majority of pharmacy students felt they learned best 
with hands-on and visual experiences in the classroom setting. 

When looking for new topics to incorporate into the spring 2012 course, student 
responses to survey question number two were assessed. The top three responses to this question 
were: evaluating information from patient profile (61.7%), locating/collecting literature & 
guidelines (56.7%), and understanding insurance & prescription drug coverage (46.7%). As a 
result, a Rounds and Patient Work-Up topic was added to teach students how to evaluate a 
patient profile prior to participating in one of many types of rounding experiences. Additionally, 
a Medicare, Medicaid, and Insurance topic was added to address deficiencies in understanding 
prescription drug coverage. The topic of identifying literature and guidelines was not added to 
the course, as this was felt to be better addressed within the Advanced Drug Information course 
as well as within other activities being incorporated throughout the curriculum. Finally, based on 
student free response comments to question number two, and Interprofessionalism topic was 
added to the course to help students understand the role and relationship of other healthcare 
professional roles within a patient care team. 

Follow-Up Survey 

The two topics that underwent revisions for the 2012 course both showed improvements 
in their ratings in the follow-up surveys (Table 1). The overall responses for the usefulness of the 
CV topic were significantly changed from 2011 to 2012 (p = 0.032). A statistically significant 
change was also seen when looking at only Useful As Is ratings for this topic (p = 0.022). In fact, 
the Curriculum Vitae topic received the most positive overall feedback of the entire course with 
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82.4% of students rating it as Useful As Is. This change was likely due the development of the 
CV topic into a more interactive, hands-on experience. Students were asked to participate in the 
lecture by providing examples of relevant experiences to include on a pharmacy student CV. 
Additionally, students were required to submit a CV to both a peer reviewer and a faculty 
reviewer in order to gain feedback. The overall ratings for the 2012 PharmD project discussion 
were not significantly changed from 2011 (p = 0.085); however, there was a statistically 
significant increase in the number of Useful As Is responses in 2012 (p = 0.024). The PharmD 
project discussions were, as suggested in student comments from the free response section, 
dispersed in small 20 to 30 minute discussions throughout the semester. The hope was to guide 
students through the natural progression of a research project and to allow students time to 
develop ideas about their own project development. 

The three topics that were added to the course based on initial survey results, Rounds and 
Patient Work-Up, Medicare/Medicaid/Insurance, and Interprofessionalism, had mixed reviews in 
the follow-up survey. 41.2% of students rated the Medicare, Medicaid, and Insurance lecture as 
Useful, Needs Improvement. One student in particular commented that this lecture “would have 
been better received if the students were engaged more.” Students were almost evenly split 
between the three ratings in terms of the Interprofessionalism lecture (35.3 % Not Useful, 32.4% 
Useful, Needs Improvement, and 32.4% Useful As Is). The Rounds and Patient Work-Up lecture 
was rated by the majority of students (58.8%) as Useful As Is. 

Table 1 

Perceived Rotation Preparation Value of Revised and New RX 500 Topics 



Not useful 

Useful, Needs 
Improvement 

Useful As-Is 

P Value 


Initial 

Follow- 

Up 

Survey 

Initial 

Follow- 

Up 

Survey 

Initial 

Follow- 

Up 

Survey 



Survey 

Survey 

Survey 


Curriculum 
Vitae Review 
PharmD 

3.3% 

2.9 % 

38.3% 

14.7% 

58.3% 

82.4% 

0.032 a 

Project 

Review 

8.3% 

2.9 % 

45.0% 

26.5% 

46.7% 

70.6% 

0.085 b 

Rounds and 
Patient 

N/A 

8.8% 

N/A 

32.4% 

N/A 

58.8% 

N/A 

Work-Up 

Medicare, 

Medicaid, 

and 

Insurance 

N/A 

23.5% 

N/A 

41.2% 

N/A 

35.3% 

N/A 

Interprofessio 

nalism 

N/A 

35.3% 

N/A 

32.4% 

N/A 

32.4% 

N/A 


a P value for only Useful As Is responses = 0.022 
h P value for only Useful As Is responses = 0.024 
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Discussion 

In this study, follow-up surveys indicated improvements in perceived student values of 
the topics that underwent revision in the 2012 course. This indicates that student perceptions of 
what is needed for clinical rotation preparation may be able to assist in creating a more effective 
course and may ultimately better provide students with skills to face the challenges of real-life 
clinical practice. New topics that were added to the course, however, had mixed reviews. The 
Medicare, Medicaid, and Insurance topic was rated by the many students as Useful, Needs 
Improvement. Additionally, although students from the initial survey were interested in learning 
about interprofessional relationships, the students in the follow-up survey did not find this topic 
to be useful in the way it was presented. This data indicate that although students are interested 
in how to interact with colleagues from different health profession disciplines, there may 
potentially be a more beneficial way to provide insight into this topic. Strategies to accomplish 
this in the future might include addition of a video demonstrating effective interactions amongst 
various health professionals, incorporation of an in-class panel of professionals who work in 
interprofessional settings, or even required shadowing experiences with a member of a working 
multidisciplinary team. 

One strength of this study is the unique study design that assessed student perceptions in 
order to shape a preparation course for clinical experiences. Students had completed at least 4 
rotation experiences prior to completing the survey, indicating a true understanding of the types 
of skills and information needed for the clinical rotation year. Additionally, this study identified 
a need for the incorporation of more hands-on experiences within the classroom setting in order 
to prepare students for the clinical experiences outside the classroom. The incorporation of this 
approach seemed to yield more favorable results during the follow-up survey period. 

Although the need for hands-on experiences within the classroom setting was identified 
and applied in this pharmacy focused study, it appears that the same teaching approach may 
provide benefits across a wide variety of health profession school settings. As was the case in 
this study, other health profession schools have evaluated the benefits of using of hands-on 
learning prior to student clinical experiences. For example, Ohtake and colleagues (2013) from 
the University of Buffalo evaluated the impact of a hands-on intensive care unit (ICU) simulation 
experience on physical therapy (PT) student confidence in caring for an ICU patient and student 
satisfaction with this learning method. The survey results revealed that PT student confidence 
increased across all areas, including the technical, behavioral, and cognitive skills required to 
care for an ICU patient. Additionally, the majority (98%) of PT students agreed or strongly 
agreed that the simulation was a valuable experience that reinforced classroom content and they 
preferred the use of simulation experiences in additional courses. 

Additionally, there are numerous studies evaluating the use of hands-on learning in 
nursing classes prior to clinical experiences (Bearnson, 2005; Comer, 2005). Beamson and 
Wiker (2005) from Brigham Young University evaluated the use of a computer-operated 
mannequin called human patient simulators (HPS) in nursing student classes to add hands-on 
experiences. The mean student survey responses following the use of the HPS experience 
revealed that students agreed that HPS increased their knowledge and confidence. Comer (2005) 
from Purdue University North Central also used the concept of HPS to create a cost effective 
alternative of clinical simulation role-playing exercises for exam topics with typically high 
failure rates in a critical care course. Following the incorporation of these hands-on simulation 
experiences, student failure rates on these exams decreased by 50%. Infonnal student feedback 
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following the exercises also showed that the majority of students found the hands-on experiences 
to be a more effective way to leam nursing concepts within the classroom. These examples from 
other health profession schools demonstrate the importance of hands-on learning prior to 
working with patients, and they support the key learning methods identified in this RX500 study. 

One of the biggest weaknesses of the study is the poor survey response during each phase 
of the study. The initial survey response rate was 50%, while the follow-up survey yielded only a 
28.8% response rate. Both of these rates are below the desired 60% response rate goal for most 
survey researchers. With the response rates seen, there is potential for selection bias of those 
individuals who had mostly positive or mostly negative perceptions of the course. Finally, there 
are likely differences in curriculum between the two classes studied. More structured placement 
of students into hospital/institutional Introductory Pharmacy Practice Experiences (IPPE) was 
introduced for the PhannD class of 2013. This additional experience may introduce differences 
in the baseline knowledge and understanding of what is needed for the APPE year. 

When evaluating existing literature across all health professional schools, it becomes 
apparent that utilizing student feedback to shape a specific didactic course and its content is a 
somewhat novel idea within health science education. This study successfully achieved that goal 
by utilizing data from an initial student survey to identify perceived areas for improvement in 
RX500’s ability to prepare pharmacy students for the APPE year. Follow-up surveys indicated 
improvements in the perceived values of the revised topics, but COPHS faculty can continue to 
work on delivery of newer topics being added to the course. Incorporation of hands-on teaching 
and learning is crucial in order to prepare health profession students prior to beginning clinical 
experiences. 
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Appendix 


Appendix 1. Initial Survey 


1. In regards to preparation for rotations, please rate the following Intro to Rotations subjects as 
’’Not Useful," "Useful, Needs Improvement,” or "Useful As Is." __ 


Topics 

Not 

Useful 

Useful, Needs 
Improvement 

Useful As 

Is 

Important 200 Medications 




Syllabus/ Schedule Review 




PharmD Project Review Day 1 




PhannD Project Review Day 2 




Financial Aid Discussion 




Community Seminar 




Professionalism 




PowerPoint Poisoning 




Drug Information Products 




CV Tune-up 
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Smoking Cessation Competency 




PhannD Manuscripts/ Abstracts/ Project 

Proposals 




Rotation Preparation-Lessons Learned 




Geriatrics Overview 




797 Preparation 




Practice Scenarios Day 1 




Practice Scenarios Day 2 




OTC Overview Day 1 




OTC Overview Day 2 




Medication Therapy Management 




Code Response 




Psych Primer 




Lessons Learned - Rotation Preparation 




Rotation Preparation Day 1 




Rotation Preparation Day 2 





Please explain items that "NeedImprovement”: 


2. Please select at least 2 subjects that would have been useful to your preparation prior to 
rotations. 

• Fulfilling rotation student expectations and responsibilities 

• Accurately and safely dispensing a medication 

• Evaluating infonnation obtained from a patient's history and physical assessment 
(general work-up, problem prioritization, etc.) 

• Identifying and collecting all information needed to prevent or resolve a medication- 
related problem 

• Calculating and evaluating drug pharmacokinetic properties for a specific patient 

• Demonstrating professionalism 

• Using appropriate behaviors and communication during professional interactions with 
other health care providers 

• Providing counseling to patients and/or caregivers relative to the proper use and 
effects of medications 

• Locating and collecting literature and guidelines 

• Understanding aspects of medical care in public health settings 

• Understanding insurance and prescription drug coverage 

Please list additional subjects that would have been useful for your rotation preparation: 

3. Which style of learning is most beneficial to you within the classroom setting? 

• Audio presentation 

• Visual presentation 

• Hand-on activity 

• Reading assignment 

• Other (please specify):_ 

4. Which of these rotations have you completed to date? 

• Ambulatory Care 

• Community Practice 
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• General Medicine 

• Institutional Practice 

• Elective (s) (please specify):_ 

5. Did you have phannacy work experience prior to starting rotations? 

• No 

• Yes (please specify number of years) 

6. If you did have pharmacy work experience prior to rotations, in what setting(s) did you 
work? 

• Hospital 

• Retail (chain) 

• Retail (independent) 

• Other (please specify):_ 
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